Foveated imaging system and method

ABSTRACT

A time-varying image is synthesized having at least one time-varying region of interest therewithin. The image is spatially modulated with an array of modulating pixels to form a modulated image. The array of modulating pixels has a modulator resolution. The modulated image is sensed with an array of sensor pixels. The array of sensor pixels has a sensor resolution. The modulator resolution is finer than the sensor resolution. Each sensor pixel corresponds to a plurality of modulating pixels. Said sensor pixel and said plurality of modulating pixels subtend the same region in the image. Outside the region of interest, the modulated image is synthesized at the sensor resolution and at a sensor video frame rate. Inside the region of interest, the modulated image is synthesized at the modulator resolution and at an effective synthesized video frame rate that is less than sensor video frame rate.

CLAIM OF PRIORITY

This patent application claims the benefit of priority of U.S.Provisional Patent Application Ser. No. 61/772,380, entitled “FoveatingCamera With Digital Micromirror Device (DMD) coding and method forproviding wide area motion imagery (WAMI) At Full-Motion Video (FMV)Rates”, filed on Mar. 4, 2013 , which is hereby incorporated byreference in its entirety.

TECHNICAL FIELD

Embodiments pertain to foveated imaging methods and systems suitable foruse on manned or unmanned intelligence, surveillance and reconnaissance(ISR) platforms.

BACKGROUND

There are many uses for collecting video imagery, such as for manned orunmanned intelligence, surveillance, and reconnaissance. For instance, acamera may be mounted in an aircraft, flown over a particular scene ofinterest, and used to collect video imagery of the scene from above.Information from the collected video imagery may be used to performvarious tasks, such as identify the presence or identity of particularvehicles or people in the scene. Particular tasks may involve differentlevels of resolution from the video imagery. For instance, a relativelycoarse resolution may allow a user to determine if a person is presentin the scene, while a relatively fine resolution may allow the user todetermine the identity of or the type of activity being performed by theperson in the scene.

SUMMARY

Many current video imaging systems are limited by data bandwidth. Suchvideo imaging systems cannot store, process, and/or transmit arelatively large stream of data produced by the simultaneous use of alarge field of view, high resolution across the full field of view, andtypical video frame rates. There exists a need for a video imagingsystem that allows for a large field of view, allows for coverage of thefield of view with a relatively coarse resolution, allows for coverageas needed of one or more regions of interest within the field of viewwith a relatively high resolution, and produces a data stream that canfit within typical data bandwidths.

An example technique, known as foveated imaging, uses an increasedresolution in a portion of the field of view. Advantageously, thepresent device and methods allow for foveated imaging using a sequenceof optical elements that can remain fixed with respect to each other,without the need for optical elements that move or can be moved withinthe field of view. The portion having increased resolution can includeone or more regions of interest. The regions of interest may be anywherewithin the field of view, and may change size and shape dynamically.

In accordance with some examples, a camera capable of providing widearea motion imagery at full-motion video rates while simultaneouslyproviding higher resolution imagery at slightly less than full-motionvideo rates in multiple regions of interest through the use of a digitalmicromirror device array in conjunction with single pixel cameracomputational imaging techniques is provided. In these examples,compressive sensing and/or single pixel camera techniques may be appliedto region(s) of a focal plane array to create one or more foveated,locally high resolution image within a full-motion video wide areamotion imagery field of view. In these examples, a digital micromirrordevice may be placed at an intermediate focal plane with a highermultiple (e.g., a 2P-by-2Q) resolution than the focal plane array (e.g.,P-by-Q). At an initial full-motion video wide area motion imagery mode,all digital micromirror device elements may be switched on to reflect afull image onto the focal plane array. Once the region of interest isspecified, a region of the digital micromirror device may be configuredto start a coded reflection pattern to capture higher resolution imagerythan the focal plane array to be reconstructed for more detail. Whilethe region of interest is being collected and reconstructed at less thanfull-motion video rate, the rest of the scene may be collected at thefull-motion video rate.

An example method synthesizes a time-varying image having at least onetime-varying region of interest therewithin. The time-varying image isspatially modulated with an array of modulating pixels to form amodulated image. The array of modulating pixels has a modulatorresolution. The modulated image is sensed with an array of sensorpixels. The array of sensor pixels has a sensor resolution. Themodulator resolution is finer than the sensor resolution. Each sensorpixel corresponds to a plurality of modulating pixels. Said sensor pixeland said plurality of modulating pixels subtend the same region in thetime-varying image. Outside the region of interest, the modulated imageis synthesized at the sensor resolution and at a sensor video framerate. Inside the region of interest, the modulated image is synthesizedat the modulator resolution and at an effective synthesized video framerate that is less than the sensor video frame rate.

In another example method, an unmodulated image is formed of atime-varying scene. The unmodulated image is modulated with amulti-pixel spatial modulator to form a modulated image of thetime-varying scene. The multi-pixel spatial modulator includes aplurality of modulator pixels. Each modulator pixel is switchablebetween an on state and an off state. Each modulator pixel in an onstate contributes to the modulated image at a location of the modulatorpixel with an optical power proportional to an optical power of theunmodulated image incident on the respective modulator pixel. Themodulated image is sensed with a multi-pixel sensor. The sensor includesa plurality of sensor pixels. The sensor pixels are fewer in number thanthe modulator pixels. The modulated image is synthesized at a firstresolution and a first video frame rate. A region of interest isidentified within the time-varying scene. The region of interest issynthesized with a second resolution finer than the first resolution,and a second video frame rate slower than the first video frame rate.

An example system images a scene. An imaging optic is configured tocollect light from the scene, to form an unmodulated incident beam fromthe collected light, and to form an unmodulated image of the scene withthe unmodulated incident beam. A spatial light modulator is configuredto receive the unmodulated incident beam, to receive the unmodulatedimage thereon, and to form a modulated exiting beam. The spatial lightmodulator includes an array of light modulating elements switchablebetween a first state and a second state. The spatial light modulatordirects portions of the unmodulated incident beam that strike lightmodulating elements in the first state into the modulated exiting beam.The spatial light modulator blocks portions of the unmodulated incidentbeam that strike light modulating elements in the second state fromentering the modulated exiting beam. A reimaging optic is configured tocollect the modulated exiting beam, and to form a spatially modulatedimage from the modulated exiting beam. The spatially modulated imageresembles the scene in areas corresponding to light modulating elementsin the first state. A sensor is configured to receive the spatiallymodulated image thereon. The sensor includes an array of sensorelements. The sensor elements are fewer in number than the lightmodulating elements. The sensor produces an electrical output signalrepresentative of the spatially modulated image. A processing element,for example a computer, is configured to receive the electrical outputsignal and switch the light modulating elements between the first stateand the second state. When at least one region of interest is identifiedin the scene, the processing element switches the light modulatingelements to produce an increase in resolution in the at least one regionof interest, and a decrease in video frame rate in the at least oneregion of interest.

This summary is intended to provide an overview of subject matter of thepresent patent application. It is not intended to provide an exclusiveor exhaustive explanation of the invention. The Detailed Description isincluded to provide further information about the present patentapplication.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, which are not necessarily drawn to scale, like numeralsmay describe similar components in different views. Like numerals havingdifferent letter suffixes may represent different instances of similarcomponents. The drawings illustrate generally, by way of example, butnot by way of limitation, various embodiments discussed in the presentdocument.

FIG. 1 is a schematic drawing of an example foveated imaging system.

FIG. 2 is a schematic drawing of an object present in a scene imaged bythe foveated imaging system of FIG. 1.

FIG. 3 is a schematic drawing of the object of FIG. 2, imaged onto thespatial light modulator of the foveated imaging system of FIG. 1, andhaving pixels from the spatial light modulator superimposed thereon.

FIG. 4 is a schematic drawing of the object of FIG. 2, imaged onto thesensor of the foveated imaging system of FIG. 1, and having pixels fromthe sensor superimposed thereon.

FIG. 5 is a schematic drawing of the object of FIG. 2, imaged onto thesensor of the foveated imaging system of FIG. 1, and having pixels fromboth the spatial light modulator and the sensor superimposed thereon.

FIG. 6 is a schematic drawing of a plurality of spatial modulatorpixels, superimposed on a corresponding single sensor pixel.

FIG. 7 is a schematic drawing of a sequence of frames for the pluralityof spatial modulator pixels of FIG. 6, with a corresponding plot overtime of an optical power on the corresponding sensor pixel.

FIG. 8 is a schematic drawing of a sequence of orthogonal frames for theplurality of spatial modulator pixels of FIG. 6, with a correspondingplot over time of an optical power on the corresponding sensor pixel.

FIG. 9 is a flow chart of an example method that can synthesize atime-varying image having at least one time-varying region of interesttherewithin.

FIG. 10 is a flow chart of an example method of operation for thefoveated imaging system of FIG. 1.

DETAILED DESCRIPTION

FIG. 1 is a schematic drawing of an example foveated imaging system 100.The system 100 is configured to receive light reflected from and/orproduced by a scene 102, where the scene 102 is not part of the system100. In an example application, the system 100 may be included within anaircraft and flown above terrain, and the scene 102 may be a portion ofthe terrain within the field of view of the imaging system 100. In mostapplications, a user does not have direct access to the scene 102, andacquires strategic information from the scene 102 only by forming imagesof the scene 102 from a distance, and interpreting the images.

The scene 102 may include strategic information, some of which may bedetermined by imaging the scene 102 with a relatively low resolution,such as the emergence of a person from a vehicle or a building. Theremay be additional strategic information that may be determined byimaging portions of the scene with a relatively high resolution, such asthe identity of a person or a license plate of a vehicle.

The system 100 can perform both low-resolution and high-resolutionimaging of the same scene 102. Typically, the system 100 performslow-resolution imaging of the entire scene 102, which occupies a fullfield of view within the system 100. The low-resolution imaging istypically performed at a standard video frame rate, which is oftenreferred to as full-motion video. Typical system bandwidths mayaccommodate the data stream produced by the simultaneous use of the fullfield of view, low resolution, and the standard video frame rate. Whenan object of interest is detected from a low-resolution image of thescene 102, such as by a user or by a software algorithm, the system 100may generate a region of interest (ROI) at or near the object ofinterest. The system can perform high-resolution imaging of the regionof interest, but at a reduced video frame rate. Typical systembandwidths may accommodate the data stream that includes thesimultaneous use of a region of interest within the field of view, highresolution, and a reduced video frame rate for the region of interest.In some examples, high resolution imaging of the region of interest maybe interleaved with low resolution imaging of the full field of view. Insome examples, more than one object of interest may be identified withina particular scene 102, and the system 100 may use two or more regionsof interest simultaneously. Regions of interest may also disappear fromthe field of view, caused by movement of the object of interest out ofthe field of view, by movement of the field of view away from the objectof interest, or by conditions within the scene 102, such as a personentering a building and disappearing from view.

The scene 102 may reflect ambient light, such as sunlight or anilluminating beam. The scene may also produce light, such as in thevisible portion of the spectrum from electric lighting, or in theinfrared portion of the spectrum, such as from people or from relativelywarm or hot objects. The system 100 collects a portion 120 of the lightfrom the scene 102. The portion 120 of light passes through an imagingoptic 104. A suitable imaging optic 104 can include a single-element ora multi-element lens, a single-element or a multi-element mirror, or acombination of at least one lens and at least one mirror. The imagingoptic 104 may include anti-reflection coatings on one or more opticsurfaces, as needed, which may be designed for suitable wavelengthranges and incident angle ranges.

The imaging optic 104 is configured to form an unmodulated incident beam122 from the collected portion 120 of the light from the scene 102. At afocal plane of the imaging optic 104, the unmodulated incident beam 122forms an unmodulated image of the scene 102. In some cases, theresolution in the unmodulated image of the scene 102 is limited bywavefront aberrations and/or chromatic aberration from the imaging optic104. For an imaging optic 104 that is well-corrected, the wavefrontaberrations and/or chromatic aberration may be sufficiently small sothat the imaging optic 104 is limited by the effects of diffraction(i.e., diffraction-limited). A typical estimate for diffraction-limitedresolution in the unmodulated image of the scene 102 is the wavelengthof the light, multiplied by the F-number of the imaging optic 104.

A spatial modulator 106 is configured to receive the unmodulatedincident beam 122. The spatial modulator 106 is spaced apart from theimaging optic 104 so that the unmodulated image is at or near thespatial modulator 106. Light exiting the spatial light modulator 106forms a modulated exiting beam 124. In most examples, the modulatedexiting beam 124 is reflected from the spatial light modulator 106,although in some examples the modulated exiting beam 124 may betransmitted through the spatial light modulator 106.

The spatial light modulator 106 includes an array of light modulatingelements 134, which may be referred to as modulator pixels. The spatiallight modulator 106 may include a rectangular array of modulator pixels,numbering M-by-N, where M and N are integers. The light modulatingelements 134 are switchable between a first state and a second state.Modulating elements 134 in the first state may direct portions of theunmodulated incident beam 122 into the modulated exiting beam 124.Modulating elements 134 in the second state may block or divert portionsof the unmodulated incident beam 122 from entering the modulated exitingbeam 124. In this manner, the spatial modulator 106 may be used tomodulate the image, on a pixel-by-pixel basis.

An example of a suitable spatial light modulator 106 is a digitalmicromirror device, or DMD. A typical DMD includes a rectangular arrayof small mirrors. Each mirror is electronically pivotable between twoangled states, the angled states being angled, for example, roughly thesame angular amount, but in opposing directions. In this example, bothangled states are typically between ten and twelve degrees away from theplane of the DMD, in opposite directions. During use, one of the twoangled states, corresponding to the first state mentioned above, directslight into the modulated exiting beam 124; this state may be consideredto be an on state. During use, the other of the two angled states,corresponding to the second state mentioned above, does not direct lightinto the modulated exiting beam 124 and typically directs light to anabsorber (not shown in FIG. 1); this state may be considered to be anoff state. Alternatively, other suitable spatial light modulators mayalso be used, including those that transmit, rather than reflect, amodulated exiting beam.

A reimaging optic 108 is configured to collect the modulated exitingbeam 124, produces a reimaged beam 126 from the modulated exiting beam124, and forms a spatially modulated image with the reimaged beam 126from the modulated exiting beam 124. Examples of suitable reimagingoptics 108 can include lenses, mirrors, and combinations of lenses andmirrors. The spatially modulated image resembles the scene 102 in areascorresponding to light modulating elements 134 in the first state, e.g.,the on state. For example, if all the light modulating elements 134 inspatial light modulator 106 were in the first state, e.g., the on state,then the spatially modulated image would look like the scene 102,possibly with an overall decrease in optical power due to losses at theimaging optic 104, the spatial light modulator 106 and the reimagingoptic 108, or due to diffraction by the spatial pattern of modulatingelements 134 in the spatial light modulator 106. As another example, ifsome of the light modulating elements 134 in spatial light modulator 106were in the second state, e.g., the off state, then the spatiallymodulated image would resemble the scene 102, but with dark locations inthe image that correspond with the locations of the off state lightmodulating elements 134 in the spatial light modulator 106.

The reimaged beam 126 strikes a sensor 110 at or near the location ofthe spatially modulated image, so that the spatially modulated image isreceived by the reimaged beam 126 thereon. The sensor 110 includes anarray of sensor elements 136. The sensor elements 136 are fewer innumber than the light modulating elements 134. The sensor elements 136are typically arranged in a rectangular array, numbering P-by-Q, where Pand Q are integers. Compared with the number of light modulatingelements 136 in the spatial light modulator 106, M-by-N, P is less thanM, Q is less than N, or both P and Q are less than M and N,respectively, such that the total number of P×Q elements 136 are lessthan the total number of M×N elements 134. The sensor 110 produces anelectrical output signal 138 representative of the spatially modulatedimage. In some examples, the sensor elements 136 are independentlyaddressable, so that the electrical output signal 138 need not scanthrough the entire set of sensor elements 136 to extract the value ofone particular element 136.

A processing element 112 is configured to receive the electrical outputsignal 138. The processing element 112 can reconstruct a video signal118 from the electrical output signal 138. The video signal 118 may bedisplayed to a user who can identify a region of interest manually,and/or may be directed to software that can automatically identify aregion of interest in the scene 102, such as a person or a vehicle. Theuser, and/or the software, may determine a current location, and/or aprojected future location of the region of interest, and may direct theregion of interest location 114 back to the processing element 112. Theprocessing element 112 may interpret the region of interest location114, and direct a suitable modulator control signal 140 to the spatiallight modulator 106. When at least one region of interest is identifiedin the scene 102, the processing element 112 switches the lightmodulating elements 134 to produce an increase in resolution in the atleast one region of interest, and a decrease in video frame rate in theat least one region of interest.

The processing element 112 may also determine a mode of operation forthe system 100, denoted as mode control 116. The processing element 112can include a general purpose computer, a tablet computing device, amobile computing device, an ASIC, a microprocessor, a FPGA, and/or thelike. The mode control 116 may determine system-based aspects, such ashow to use the region of interest, whether the system is in a losslessor a compressive sensing mode, and so forth. For instance, in a losslessmode, pixel cycling schemes, such as those discussed with reference toFIGS. 8 and 9, may be applied only to the region(s) of interest; in acompressive sensing mode, different schemes known in the art may beapplied only to the region(s) of interest. In either mode, no modulationis applied to the portions of the scene 102 that are outside theregion(s) of interest.

An example object 128 within the scene 102 is shown schematically as anarrow in FIG. 1. The example object 128 appears in both the unmodulatedimage and the modulated image in the system 100, where they are alsoshown as arrows in FIG. 1. The unmodulated image at the spatial lightmodulator 106 includes an arrow 130. The arrow 130 at the spatial lightmodulator 106 is inverted, compared to the arrow 128 in the scene 102,due to inverting properties of the imaging optic 104. The modulatedimage at the sensor 110 also includes an arrow 132. The arrow 132 isalso inverted, due to inverting properties of the reimaging optic 108.

During operation, the imaging optic 104, the spatial light modulator106, the reimaging optic 108, and the sensor 110 all remain in fixedpositions with respect to one another. A particular location within thescene 102 maps to a unique location on the spatial light modulator 106,and a corresponding unique location on the sensor 110. As a result, onemay draw correspondences between the light modulating elements 134 inthe spatial light modulator 106 and the pixels in the sensor 110. Iflight from the scene 102 strikes a particular light modulating element134 (in the on state), then the light will proceed through the system100 to strike the pixel on the sensor 110 that corresponds to theparticular light modulating element 134. FIGS. 2-5 show an example setof correspondences among an object in the scene, the light producingelements in the spatial light modulator, and the pixels in the sensor.

FIG. 2 is a schematic drawing of the object 202, present in the scene102, and imaged by the foveated imaging system 100 of FIG. 1. Althoughthe object 202 is drawn as an arrow, the object may be any suitableobject of interest within the scene 102, such as a person or a vehicle.

FIG. 3 is a schematic drawing of the object 202 of FIG. 2, imaged ontothe spatial light modulator 106 of the foveated imaging system 100 ofFIG. 1, and having pixels from the spatial light modulator 106superimposed thereon. FIG. 3 includes a grid 304 showing the boundariesbetween adjacent light modulating elements 134 in the spatial lightmodulator 106. The grid 304 is superimposed on the representation 302 ofthe object in the unmodulated image.

FIG. 4 is a schematic drawing of the object 202 of FIG. 2, imaged ontothe sensor 110 of the foveated imaging system 100 of FIG. 1, and havingpixels from the sensor 110 superimposed thereon. FIG. 4 includes a grid206 showing the boundaries between adjacent pixels in the sensor 110.The grid 406 is superimposed on the representation 402 of the object inthe modulated image.

FIG. 5 is a schematic drawing of the object 202 of FIG. 2, imaged ontothe sensor 110 of the foveated imaging system 100 of FIG. 1, and havingpixels from both the spatial light modulator 106 and the sensor 110superimposed thereon. FIG. 5 includes a dashed grid 504 showing theboundaries between adjacent light modulating elements 134 in the spatiallight modulator 106. FIG. 5 also includes a solid grid 506 showing theboundaries between adjacent pixels in the sensor 110. Both grids 504,506 are superimposed on the representation 502 of the object in themodulated image.

As noted above, the sensor 110 includes fewer pixels than the spatiallight modulator 106. In the example of FIGS. 2-5, the spatial lightmodulator 106 has twice as many pixels as the sensor 110, in bothorthogonal directions, so that each pixel on the sensor 110 correspondsto four pixels on the spatial light modulator 106. This four-to-oneratio is just one example, and other suitable ratios may also be used.

Because the pixels of the spatial light modulator 106 and the sensor 110remain in a fixed relationship to each other during operation, it isbeneficial to examine a single sensor pixel and the corresponding pixelson the spatial light modulator. FIG. 6 is a schematic drawing of aplurality of spatial modulator pixels 610, 612, 614, 616, superimposedon a corresponding single sensor pixel 606. The amount of optical powerin each of the spatial modulator pixels 610, 612, 614, 616 isrepresented by A, B, C, and D, respectively. Grid lines 604 representthe boundaries between adjacent pixels on the spatial modulator.

This fixed relationship between a sensor pixel 606 and its correspondingspatial modulator pixels 610, 612, 614, 616 may be used to providevarying degrees of resolution for the scene 102, as the scene 102 variesin time. For instance, if all the light modulating elements 134 are setto the on state, then the sensor pixel 606 receives light from all ofits corresponding spatial modulator pixels 610, 612, 614, 616, and thesensor pixel 606 records an optical power value equal to (A+B+C+D). Thisexample represents relatively low resolution of the scene 102, becauseoptical features in the scene smaller than a sensor pixel 606 are notresolved.

An example representing relatively high resolution is shown in FIG. 7.FIG. 7 is a schematic drawing of a sequence 700 of frames 706A-H for aplurality of spatial modulator pixels 702 associated with a singlesensor pixel. In this example, only one modulator pixel 702 at a time isin the on state, while the other modulator pixels 702 in the pluralityare in the off state. If there are X number of modulator pixels 702 inthe plurality, then the sequence 700 includes X frames, with each framehaving a different modulator pixel 702 in the on state.

FIG. 7 also includes a corresponding plot 704 over time of an opticalpower on the sensor pixel. The sequence of power levels corresponds tothe optical power striking each of the modulator pixels 702. Aparticular optical power value may be obtained from an electrical signalfrom the sensor pixel, from examination of the electrical signal at aparticular time. The example of FIG. 7 represents relatively highresolution of the scene 102, because optical features in the scenecomparable to a modulator pixel 702 may be resolved. Compared with thelow resolution example, the high resolution example in FIG. 7 may takelonger to transmit over a data channel having a fixed data rate, by afactor of X, where X is the number of modulator pixels 702 in theplurality. Typical systems may use this high resolution only for one ormore regions of interest within the scene 102. Typical systems maytransmit the high resolution data corresponding to the one or moreregions of interest at a reduced video frame rate, so as not to exceed amaximum data rate for transmission, storage and/or processing theelectrical signals from the sensor pixels.

Another example representing relatively high resolution is shown in FIG.8. FIG. 8 is schematic drawing of a sequence 800 of orthogonal frames806A-H for the plurality of spatial modulator pixels 802 associated witha single sensor pixel. In this example, more than one spatial modulatorpixels 802 at a time is in the on state. If there are X number ofmodulator pixels 802 in the plurality, then the sequence 800 includes Xframes, with each frame having a different combination of modulatorpixels 802 in the on state. The frames 806A-H are said to be orthogonalbecause the scene information content of any single frame is devoid ofinformation from the other frames in the sequence. Other suitableorthogonal sequences may be used.

FIG. 8 also includes a corresponding plot 804 over time of optical poweron the sensor pixel. In this example, the optical power on the sensorpixel cycles through four combinations of modulator pixels, shown in thefour frames 806A-D and repeated in 806E-H. For convenience, the fourcombinations of modulator pixels are denoted as R, S, T, and U, and aredefined as follows: R=(A+B+C+D), S=(A+C), T=(A+B), and U=(A+D). Theindividual power levels for A, B, C, and D may be calculated from thefour power levels on the sensor pixel as follows: A=(−R+S+T+U)/2,B=(R−S+T−U)/2, C=(R+S−T−U)/2, and D=(R−S−T+U)/2.

In the examples of FIGS. 7 and 8, the effective synthesized video framerate is one-fourth of the sensor video frame rate, and is alsoone-fourth of the rate at which the modulator pixels are refreshed. Insome cases, a high-resolution image may be delivered for every fourvideo frames. For these cases, in the example of FIG. 8, the pixelvalues of (R,S,T,U) may be used for each high-resolution image, with apause between each high-resolution image to refresh all four values ofR, S, T, and U. In other cases, it may be desirable to deliver an imageat the full video frame rate. For these cases, only one of the fourpixel values is refreshed from frame-to-frame. For instance, a firstimage may use the sequence of (R,S,T,U), the next image may use thesequence of (S,T,U,R), then the next image may use (T,U,R,S), and soforth. For these cases, the synthesized image is refreshed every frame,but maintains a three-frame latency for each image.

The signal to noise ratios of the examples of FIGS. 7 and 8 are expectedto be the same, assuming that each frame has the same readout noise, andthe readout noise terms add in quadrature. In general, it is expectedthat the signal to noise ratio should be the same for most, or all,orthogonal modulation sequences. In some applications, the modulationscheme of FIG. 8 may be preferable to that of FIG. 7, because thedynamic range of the sensor and the associated electronic components maybe reduced. Furthermore, if low resolution imaging is interspersed withhigh resolution imaging, then it may be preferable to use a modulationscheme that includes a frame that has all the modulation pixels set toan on state, as in 806A, because such a modulation scheme may allow foruse of the same frame (e.g., 806A) for both the low resolution imagingand the high resolution imaging, thereby optimizing the frame rate forboth low resolution and high resolution.

It is beneficial to clarify the terms sensing and synthesizing, withrespect to a particular image. The term sensing, as used herein, isintended to mean the process of physically acquiring modulated imageryat the sensor 110, which is a physical light-sensing detector. Sensing,as used herein, occurs at the same rate as the modulation at the spatiallight modulator 106. The term synthesizing, as used herein, is intendedto mean forming an image from data corresponding to one or more regionsof interest within the scene. The mapping schemes of FIGS. 7 and 8 areexamples of how a sensor image may be synthesized, pixel-by-pixel, froma corresponding modulator image.

To further clarify, the term effective synthesized video frame rate isintended to mean the rate at which the sensor image is synthesized. Inthe examples of FIGS. 7 and 8, the effective synthesized video framerate is one-fourth of the rate at which the modulator pixels arerefreshed. In these examples, the effective synthesized video frame rateis also one-fourth of the sensor video frame rate.

The effective rate at which the processing element 112 synthesizesimages of the region(s) of interest at the full modulator resolution maybe lower than at the full sensing rate, due to latency in acquiring thesequence of images, but the full modulator resolution images may stillbe updated at the full sensing rate. For instance, if a scene changesrelatively slowly, the sub-images within the region of interest mayappear to move at the full sensing rate. If the scene changes relativelyquickly, the images may be distorted due to the latency, but the imagesmay change with the rest of the scene.

FIG. 9 is a flow chart of an example method 900 that can synthesize atime-varying image having at least one time-varying region of interesttherewithin. Step 902 spatially modulates the time-varying image with anarray of modulating pixels to form a modulated image. A suitable arrayof modulating pixels may be spatial light modulator 106 of FIG. 1. Thearray of modulating pixels has a modulator resolution. Step 904synthesizes the modulated image with an array of sensor pixels. Asuitable array of sensor pixels may be the sensor 110 of FIG. 1. Thearray of sensor pixels has a sensor resolution. The modulator resolutionis finer than the sensor resolution. Each sensor pixel corresponds to aplurality of modulating pixels. Said sensor pixel and said plurality ofmodulating pixels subtend the same region in the time-varying image.Outside the region of interest, the modulated image is synthesized atthe sensor resolution and at a sensor video frame rate. Inside theregion of interest, the modulated image is synthesized at the modulatorresolution and at an effective synthesized video frame rate that is lessthan the sensor video frame rate.

In addition, each modulating pixel may be switchable between an on stateand an off state. The on state may comprise directing at least a portionof time-varying image light, incident on said modulating pixel, to themodulated image. The off state may comprise blocking the time-varyingimage light, incident on said modulating pixel. Inside the region ofinterest, said plurality of modulating pixels may be modulated in anorthogonal sequence of frames. In one frame of the orthogonal sequence,all the modulating pixels in the plurality may be in the on state. Inall frames of the orthogonal sequence, at least two of the modulatingpixels in the plurality may be in the on state.

FIG. 10 is a flow chart of an example method 1000 of operation for thefoveated imaging system 100 of FIG. 1. Step 1002 forms an unmodulatedimage of a time-varying scene. Step 1004 modulates the unmodulated imagewith a multi-pixel spatial modulator to form a modulated image of thetime-varying scene. The multi-pixel spatial modulator includes aplurality of modulator pixels. Each modulator pixel is switchablebetween an on state and an off state. Each modulator pixel in an onstate contributes to the modulated image at a location of the modulatorpixel with an optical power proportional to an optical power of theunmodulated image incident on the respective modulator pixel. Step 1006synthesizes the modulated image with a multi-pixel sensor. The sensorincludes a plurality of sensor pixels. The sensor pixels are fewer innumber than the modulator pixels. The modulated image is synthesized ata first resolution and a first video frame rate. Step 1008 identifies aregion of interest within the time-varying scene. Step 1010 synthesizesthe region of interest with a second resolution finer than the firstresolution, and a second video frame rate slower than the first videoframe rate.

The first resolution may correspond to the resolution of the multi-pixelsensor. The second resolution may correspond to the resolution ofmulti-pixel spatial modulator. Each sensor pixel may correspond with aplurality of the modulator pixels. Said sensor pixel and said pluralityof modulator pixels may occupy the same position within the time-varyingscene. Within the region of interest, said plurality of modulator pixelsmay be switched on and off in an orthogonal sequence. The number ofmodulator pixels in said plurality may be X. Within the region ofinterest, said sensor pixel may record X values of optical power. The Xvalues of optical power may be orthogonal combinations of optical powerfrom the X modulator pixels. The optical powers of the individualmodulator pixels may be determined from the orthogonal combinations ofoptical power from the modulator pixels. In one of the orthogonalcombinations, all the modulator pixels in the plurality may be switchedon. The optical powers of the individual modulator pixels may bedetermined at second video frame rate. The second video frame rate mayequal the first video frame rate, divided by X. Outside the region ofinterest, said plurality of modulator pixels may be all switched on.Outside the region of interest, the optical powers of the individualmodulator pixels may be combined onto the same sensor pixel and may beindistinguishable from one another. The unmodulated image of thetime-varying scene may be formed with an imaging optic disposed betweenthe time-varying scene and the unmodulated image. The modulated image ofthe time-varying scene may be formed with a reimaging optic disposedbetween the multi-pixel spatial modulator and the modulated image. Eachof the above statements may be combined with any of the other statementsin any combination.

Some embodiments may be implemented in one or a combination of hardware,firmware and software. Embodiments may also be implemented asinstructions stored on a computer-readable storage device, which may beread and executed by at least one processor to perform the operationsdescribed herein. A computer-readable storage device may include anynon-transitory mechanism for storing information in a form readable by amachine (e.g., a computer). For example, a computer-readable storagedevice may include read-only memory (ROM), random-access memory (RAM),magnetic disk storage media, optical storage media, flash-memorydevices, and other storage devices and media. In some embodiments,system 100 may include one or more processors and may be configured withinstructions stored on a computer-readable storage device.

What is claimed is:
 1. A method of sensing a time-varying image havingat least one time-varying region of interest therewithin, the methodcomprising: spatially modulating the time-varying image with an array ofmodulating pixels to form a modulated image, the array of modulatingpixels having a modulator resolution; and sensing the modulated imagewith an array of sensor pixels, the array of sensor pixels having asensor resolution, the modulator resolution being finer than the sensorresolution; wherein each sensor pixel corresponds to a plurality ofmodulating pixels, said sensor pixel and said plurality of modulatingpixels subtending the same region in the time-varying image; whereinoutside the region of interest, the modulated image is synthesized atthe sensor resolution and at a sensor video frame rate; wherein insidethe region of interest, the modulated image is synthesized at themodulator resolution and at an effective synthesized video frame ratethat is less than the sensor video frame rate; wherein each modulatingpixel is switchable between an on state and an off state; wherein the onstate comprises directing at least a portion of time-varying imagelight, incident on said modulating pixel, to the modulated image;wherein the off state comprises blocking the time-varying image light,incident on said modulating pixel; wherein inside the region ofinterest, said plurality of modulating pixels are modulated in anorthogonal sequence of frames; and wherein in one frame of theorthogonal sequence, all the modulating pixels in the plurality are inthe on state.
 2. A method of sensing a time-varying image having atleast one time-varying region of interest therewithin, the methodcomprising: spatially modulating the time-varying image with an array ofmodulating pixels to form a modulated image, the array of modulatingpixels having a modulator resolution; and sensing the modulated imagewith an array of sensor pixels, the array of sensor pixels having asensor resolution, the modulator resolution being finer than the sensorresolution; wherein each sensor pixel corresponds to a plurality ofmodulating pixels, said sensor pixel and said plurality of modulatingpixels subtending the same region in the time-varying image; whereinoutside the region of interest, the modulated image is synthesized atthe sensor resolution and at a sensor video frame rate; wherein insidethe region of interest, the modulated image is synthesized at themodulator resolution and at an effective synthesized video frame ratethat is less than the sensor video frame rate; wherein each modulatingpixel is switchable between an on state and an off state; wherein the onstate comprises directing at least a portion of time-varying imagelight, incident on said modulating pixel, to the modulated image;wherein the off state comprises blocking the time-varying image light,incident on said modulating pixel; wherein inside the region ofinterest, said plurality of modulating pixels are modulated in anorthogonal sequence of frames; and wherein in all frames of theorthogonal sequence, at least two of the modulating pixels in theplurality are in the on state.
 3. A method, comprising: forming anunmodulated image of a time-varying scene; modulating the unmodulatedimage with a multi-pixel spatial modulator to form a modulated image ofthe time-varying scene, the multi-pixel spatial modulator including aplurality of modulator pixels, each modulator pixel being switchablebetween an on state and an off state; each modulator pixel in an onstate contributing to the modulated image at a location of the modulatorpixel with an optical power proportional to an optical power of theunmodulated image incident on the respective modulator pixel; sensingthe modulated image with a multi-pixel sensor, the sensor including aplurality of sensor pixels, the sensor pixels being fewer in number thanthe modulator pixels, the modulated image being synthesized at a firstresolution and a first video frame rate; identifying a region ofinterest within the time-varying scene; and synthesizing the region ofinterest with a second resolution finer than the first resolution, and asecond video frame rate slower than the first video frame rate; whereineach sensor pixel corresponds with a plurality of the modulator pixels,said sensor pixel and said plurality of modulator pixels occupying thesame position within the time-varying scene; wherein within the regionof interest, said plurality of modulator pixels are switched on and offin an orthogonal sequence; wherein the number of modulator pixels insaid plurality is X; wherein within the region of interest, said sensorpixel records X values of optical power, the X values of optical powerbeing orthogonal combinations of optical power from the X modulatorpixels; wherein the optical powers of the individual modulator pixelsare determined from the orthogonal combinations of optical power fromthe modulator pixels; and wherein in one of the orthogonal combinations,all the modulator pixels in the plurality are switched on.
 4. A method,comprising: forming an unmodulated image of a time-varying scene;modulating the unmodulated image with a multi-pixel spatial modulator toform a modulated image of the time-varying scene, the multi-pixelspatial modulator including a plurality of modulator pixels, eachmodulator pixel being switchable between an on state and an off state;each modulator pixel in an on state contributing to the modulated imageat a location of the modulator pixel with an optical power proportionalto an optical power of the unmodulated image incident on the respectivemodulator pixel; sensing the modulated image with a multi-pixel sensor,the sensor including a plurality of sensor pixels, the sensor pixelsbeing fewer in number than the modulator pixels, the modulated imagebeing synthesized at a first resolution and a first video frame rate;identifying a region of interest within the time-varying scene; andsynthesizing the region of interest with a second resolution finer thanthe first resolution, and a second video frame rate slower than thefirst video frame rate; wherein each sensor pixel corresponds with aplurality of the modulator pixels, said sensor pixel and said pluralityof modulator pixels occupying the same position within the time-varyingscene; wherein outside the region of interest, said plurality ofmodulator pixels are all switched on; and wherein outside the region ofinterest, the optical powers of the individual modulator pixels arecombined onto the same sensor pixel and are indistinguishable from oneanother.